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BY  THE  COMPTROLLER  GENERAL  ~ 


Report  To  The  Congress 

OF  THE  UNITED  STATES 


Better  Planning  And  Management  Of  Threat 
Simulators  And  Aerial  Targets  Is  Crucial 
To  Effective  Weapon  Systems  Performance 


Operational  test  and  evaluation  is  the  most 
effective  way,  under  peacetime  conditions, 
to  determine  a  weapon  system's  combat 
worth.  Realistic  testing,  however,  requires 
test  resources,  such  as  threat  simulators 
and  aerial  targets,  that  duplicate,  to  the 
extent  possible,  the  characteristics  of  the 
enemy's  weapon  systems. 

DOD's  test  resource  planning  practices, 
organizational  structures,  management  em¬ 
phasis,  and  intelligence  support  for  threat 
simulators  and  aerial  targets  can  be  im¬ 
proved  to  provide  better  assurance  that  cur¬ 
rent  and  future  weapon  systems  will  meet 
performance  expectations. 
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To  the  President  of  the  Senate  and  the 
Speaker  of  the  House  of  Representatives 

This  report  addresses  the  Department  of  Defense's  capability 
to  test  its  weapon  systems  and  recommends  actions  that  will 
improve  the  completeness  and  adequacy  of  weapon  testing.  This 
report  focuses  on  testing  against  two  of  the  more  significant 
threats  to  the  military  services — the  newer  Soviet  aircraft  and 
missile  systems  and  the  Soviet's  considerable  capabilities  in 
electronic  warfare. 

We  are  sending  copies  of  this  report  to  the  Secretary  of 
Defense  and  to  the  Director,  Office  of  Management  and  Budget. 

Comptroller  General  / 
of  the  United  States 
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BETTER  PLANNING  AND  MANAGEMENT 
OF  THREAT  SIMULATORS  AND  AERIAL 
TARGETS  IS  CRUCIAL  TO  EFFECTIVE 
WEAPON  SYSTEMS  PERFORMANCE 

DIGEST 

Under  peacetime  conditions,  operational  test¬ 
ing  in  a  realistic  environment  against  equip¬ 
ment  that  simulates  the  enemy's  weapons  con¬ 
tinues  to  be  the  most  credible  demonstration 
of  the  combat  worth  of  our  weapon  systems. 

All  three  military  services,  however,  are 
unable  to  test,  to  the  extent  possible,  many 
of  their  weapon  systems  in  a  representative 
combat  environment.  This  is  because  develop¬ 
ment  of  electronic  warfare  threat  simulators 
and  aerial  targets  have  not  kept  pace  with  the 
deployment  of  the  enemy's  weapons.  For 
example,  the  United  States'  inventory  of 
threat  simulators  does  not  cover  many  of  the 
enemy's  weapon  systems  that  our  weapons  may 
face.  Some  threat  systems,  such  as  certain 
radar  jammers,  have  no  simulator  coverage  at 
all. 


As  a  result  of  the  shortfalls,  important 
weapons — such  as  the  Navy’s  Aegis  cruiser  and 
improved  Phoenix  air-to-air  missiles,  the 
Army's  Patriot  missile  system,  and  the  Air 
Force's  new  B-52  offensive  avionics  system, 
will  be  deployed  without  having  fully  demon¬ 
strated  their  capabilities  under  representa¬ 
tive  combat  conditions.  The  DOD  has  recogniz¬ 
ed  the  importance  of  electronic  warfare  threat 
simulators  and  aerial  targets  and  has  describ¬ 
ed  overcoming  the  deficiencies  as  challenges. 
(See  ch.  2.) 

Test  resources  include  facilities,  equipment, 
and  personnel.  In  this  review  GAO  focused  on 
two  types — electronic  warfare  threat  simula¬ 
tors  and  aerial  targets.  GAO  concentrated  on 
these  two  resources  because  of  their  impor¬ 
tance  in  demonstrating  weapon  system  perfor¬ 
mance  and  because  of  congressional  concern 
over  the  ability  of  weapon  systems  to  meet 
their  intended  missions.  (See  pp.  2  and  3.) 

GAO  found  that  shortcomings  in  electronic  war¬ 
fare  threat  simulators  and  aerial  targets 
were  primarily  caused  by  problems  in  four 
areas.  These  areas  are 
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—  test  resource:  planning, 

— organizational  structures, 

— management  emphasis,  and 

—  intelligence  support. 

GAO  found  that  the  problems  in  planning  and 
management  emphasis  are  not  limited  to  threat 
simulators  and  aerial  targets,  but  also  cause 
problems  in  other  test  resources.  (See 
ch.  3.) 

Resolving  issues  in  these  four  areas  will  not 
necessarily  solve  all  of  the  problems.  For 
example,  legitimate  safety  and  environmental 
concerns  restrict  the  use  of  certain  test 
resources  and  thereby  constrain  realistic 
testing.  However,  as  addressed  below,  GAO 
believes  improvements  in  planning,  organiza¬ 
tion,  management  emphasis,  and  intelligence 
support  will  ultimately  result  in  better 
tested  and  thus,  more  effective  weapons  for 
our  fighting  forces.  (See  pp.  6  and  13.) 

TEST  RESOURCE  PLANNING 

Early  and  detailed  test  planning  is  needed  to 
allow  (1)  timely  identification  of  test 
resource  needs,  (2)  time  necessary  for  devel¬ 
oping  needed  resources,  and  (3)  disclosure  of 
critical  testing  issues  so  that  risks  can  be 
sufficiently  identified  to  decisionmakers. 

This  planning  is  now  done  through  test  and 
evaluation  master  plans  prepared  by  the  weapon 
systems  developer  and  the  operational  test 
agency. 

GAO's  review  of  selected  weapon  systems  and 
the  Department  of  Defense  (DOD)  studies  show 
that  test  and  evaluation  master  plans  have  not 
been  detailed  or  timely  and,  therefore,  have 
not  provided  the  test  resource  developer  with 
sufficient  information  or  time  to  ensure  that 
test  equipment  is  available  when  weapon 
systems  are  ready  for  testing.  For  example, 
the  EF-111A  tactical  jamming  system  test 
requirements  were  made  known  to  the  test 
facility  only  4  months  before  testing  was 


to  begin.  The  facility  did  not  have  the 
necessary  resources  and  could  not  obtain  them 
in  time.  Consequently,  only  limited  testing 
was  accomplished.  (See  p.  14.) 

ORGANIZATIONAL  STRUCTURES 

GAO  found  that  known  organizational  problems 
continue  to  exist  in  the  Air  Force  and  the  Navy 
and  are  impeding  the  acquisition  of  electronic 
warfare  and  aerial  target  test  resources.  The 
Army,  however,  has  corrected  its  organizational 
problems  through  reorganization.  The  Navy's 
organization  for  electronic  warfare,  as  an 
example  of  inappropriate  organization,  is  frag¬ 
mented  with  11  different  types  of  Navy  command 
organizations  having  overlapping  responsibili¬ 
ties  in  electronic  warfare  testing. 

These  organizational  arrangements  often  force 
weapon  developers  to  choose,  for  example, 
between  delaying  the  development  of  a  missile 
or  the  threat  simulator  used  to  evaluate  the 
effectiveness  of  that  missile.  In  one  case,  a 
Navy  office  that  develops  electronic  warfare 
equipment  also  develops  the  equipment  used  to 
test  and  evaluate  the  combat  effectiveness  of 
the  electronic  warfare  devices.  Similarily, 
the  Air  Force's  Firebolt  aerial  target  is  under 
the  same  management  structure  as  weapons  such 
as  the  Advanced  Medium  Range  Air-to-Air  Missile 
and  thus  competes  for  funds,  priority,  and 
personnel.  While  these  trade-offs  between 
weapon  systems  and  test  resources  may  be 
necessary,  they  should  be  made  at  service 
headquarters  levels  as  they  are  in  the  Army, 
where  a  broader  view  of  total  force 
requirements  can  be  applied.  (See  p.  16.) 

MANAGEMENT  EMPHASIS 

GAO  believes  that  insufficient  support  for 
testing  and  test  resources  within  the  DOD 
weapon  development  community  is  contributing 
to  inadequate  and,  in  some  cases,  nonexistent 
test  resources.  Important  test  resources  such 
as  the  Firebolt  aerial  target  and  electronic 
warfare  threat  simulators  have  not  been  ade¬ 
quately  funded  because  top  management,  in  many 
cases,  lacks  '■he  commitment  necessary  to  ensure 
the  timely  development  of  test  resources.  Many 
internal  service  studies  have  carried  this  mes¬ 
sage.  For  example,  in  a  review  of  the  adequacy 
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of  electronic  warfare  test  capability,  a  1982 
Air  Force  study  stated  that  electronic  warfare 
test  resource  capability  improvements  have  not 
been  available  when  needed,  primarily  because 
the  improvements  have  not  been  funded.  Discus¬ 
sions  with  top  service  and  the  Office  of  the 
Secretary  of  Defense  (OSD)  officials  corrob¬ 
orate  the  studies.  For  example,  the  former 
Director  of  the  Navy's  Electronic  Warfare  Divi¬ 
sion  stated  the  emphasis  is  on  deploying  sys¬ 
tems  first  and  then  testing  them.  (See  p.  19.) 

INTELLIGENCE  SUPPORT 

GAO  found  that  several  threat  simulators  have 
been  developed  using  inaccurate  and  incomplete 
intelligence  data.  Testing  weapons  with  non¬ 
representative  threat  simulators  further  adds 
to  the  problems  of  achieving  realism  in  testing 
and  reduces  confidence  in  the  accuracy  and  use¬ 
fulness  of  the  test  results. 

The  Defense  Audit  Service  and  other  DOD  agen¬ 
cies  have  also  reviewed  intelligence  support  to 
test  and  evaluation;  each  has  found,  as  GAO 
did,  significant  problems.  The  identified 
problems  range  from  intelligence  data  being  too 
sensitive  to  disseminate,  to  inappropriate 
assignment  of  data  analysis  responsibilities  by 
the  Defense  Intelligence  Agency.  GAO  believes, 
however,  that  most  of  the  recommendations  in 
the  DOD  studies  do  not  adequately  address  the 
underlying  causes  of  what  is  generally  per¬ 
ceived  as  poor  intelligence  support  to  test  and 
evaluation.  GAO  also  believes  that  further 
evaluation  by  DOD  is  necessary  to  better  define 
the  problems  and  develop  appropriate  solu¬ 
tions.  (See  p.  21.) 

EFFORTS  TO  IMPROVE  TEST  CAPABILITIES 

Recent  DOD  initiatives  to  improve  test  capa¬ 
bilities  are  encouraging.  The  Under  Secretary 
of  Defense,  Research  and  Engineering,  in  a  fis¬ 
cal  year  1984  budget  statement  to  the  Congress, 
discussed  threat  simulator  and  aerial  target 
deficiencies  and  characterized  them  as  chal¬ 
lenges.  In  addition,  the  Army  has  reorganized 
its  threat  simulator  development  program;  the 
Navy  has  recognized  its  fragmented  management 
of  electronic  warfare;  the  Air  Force  is  making 


changes  in  its  fragmented  management  struc¬ 
ture;  and  OSD  has  alerted  its  top  management 
to  the  threat  simulator  problems.  Most  of 
these  efforts,  however,  identify  problem  areas 
but  do  not  offer  definite  plans  for  corrective 
action.  GAO  is  encouraged  by  these  latest 
efforts,  but  recognizes  that  they  are  first 
steps  and  should  be  followed  by  defini¬ 
tive  action  plans.  (See  pp.  1,  17,  18,  19, 
and  23 .  ) 

AGENCY  COMMENTS 

Representatives  of  OSD,  the  Army,  Navy,  and 
Air  Force  orally  commented  on  a  draft  of  this 
report.  They  were  primarily  concerned  that 
GAO  had  over  generalized  its  findings  rather 
than  limiting  them  to  electronic  warfare 
threat  simulators  and  aerial  targets.  In 
those  instances  where  GAO's  findings,  conclu¬ 
sions,  and  recommendations  apply  only  to  elec¬ 
tronic  warfare  threat  simulators  and  aerial 
targets,  GAO  changed  the  report  accordingly. 
However,  since  this  and  prior  GAO  reports  show 
that  shortcomings  in  planning  and  management 
emphasis  apply  generally  to  testing  and  test 
resources-,  GAO  believes  changes  in  these  areas 
are  not  necessary.  (See  p.~  24.) 

Overall,  DOD  believes  that  its  testing  program 
and  test  resources  are  adequate.  DOD  states 
further  that  it  is  committed  to  a  continuing, 
sound  test  resources  program.  DOD  also  stated 
that  GAO's  constructive  suggestions  in  threat 
simulator  and  aerial  target  test  resources  are 
welcome  and  that  DOD  will  continue  to  empha¬ 
size  those  areas.  (See  p.  25.) 

DOD  disagreed,  however,  with  GAO's  findings 
regarding  the  completeness  of  testing,  the  Air 
Force's  management  of  aerial  targets  and 
threat  simulators,  and  inadequate  intelligence 
support  to  the  test  community.  (See  p.  26.) 

DOD  also  disagreed  with  GAO's  proposals  to 
strengthen  and  enforce  the  test  planning  proc¬ 
ess  as  well  as  to  closely  review  the  issues 
surrounding  intelligence  support  to  test  and 
evaluation.  In  general,  DOD  stated  that  their 
existing  practices  and  policies  address  the 


problems  idem  i  r  ( .  ■.  ,  w .  However,  GAO 

pointed  out  tre-  i  t  showed  that 

these  efforts  have  ...  .r.  oeen  fully  effective 
and  further  actio...-.  are  seeded.  in  addition, 
the  Secretary  of  defense,  in  his  fiscal  year 
1984  budget  statement  to  the  Congress  recog¬ 
nized  that  deficiencies  exist  in  threat  simu¬ 
lators  and  aerial  targets  and  called  for  a 
program  to  improve  the  realism  in  testing. 

(See  pp .  24  to  2  6 . } 

RECOMMENDATIONS  TO 

THE  SECRETARY  OF  DEFENSE 

To  improve  the  planning  and  management  of 
critical  test  resources,  GAO  recommends  that 
the  Secretary  of  Defense: 

— Take  several  actions  to  improve  the  timeli¬ 
ness,  completeness,  and  usefulness  of  the 
Test  and  Evaluation  Master  Plans. 

(See  p.  26 . ) 

—  Improve  the  Navy's  and  the  Air  Force's 
organizations  for  developing  threat  simu¬ 
lators  ana  aerial  targets  by  transferring 
acquisition  responsibility  to  a  service 
organization  with  the  independence,  author¬ 
ity,  responsibility,  and  funds  to  ensure 
successful  development  of  test  resources. 

( See  p .  26  .  ) 

—  Initiate  a  review  of  intelligence  support  to 
identify  the  underlying  causes  and  to  solve 
the  problems  of  inadequate  support  to  the 
threat  simulator  development  community. 

( See  p .  26.) 

Because  of  the  magnitude  of  threat  simulator 
and  aerial  target  shortfalls,  GAO  believes 
there  is  an  immediate  need  for  a  DOD-wide  pro¬ 
gram  to  improve  the  situation.  GAO  therefore, 
also  recommends  that  the  secretary  of  Defense 
establish  a  joint-service  threat  simulator  and 
aerial  target  improvement  program  to  priori¬ 
tize,  time  pnase,  ami  resolve  DOD-wide  test 
resource  problems.  (See  p.  26.) 


RECOMMENDATION  TO  THE  CONGRESS 


This  report  shows  that  major  weapon  systems 
are  being  deployed  without  the  best  possible 
demonstration  of  their  capabilities  under  com¬ 
bat  type  conditions.  DOD  has  been  giving  the 
Congress  reports  on  the  cost,  schedule,  and 
performance  status  of  its  major  weapon  systems 
since  1969.  Additionally,  annual  hearings  on 
weapon  systems  have  continually  highlighted 
weapon  performance  issues  and  the  need  for 
better  test  and  evaluation,  as  have  many  GAO 
reports.  Nevertheless,  the  problems  associ¬ 
ated  with  inadequate  and  incomplete  testing 
continue . 

Since  improvements  in  test  resources  will  lead 
to  better  testing  and  thus,  to  better  weapon 
system  performance,  GAO  recommends  the  Con¬ 
gress  monitor  DOD's  implementation  of  (1)  the 
joint-service  aerial  target  and  electronic 
warfare  test  resource  improvement  program  and 
(2)  the  separation  of  test  resource  and  weapon 
system  acquisition  management  in  the  Navy  and 
the  Air  Force.  (See  p.  27.) 
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CHAPTER  1 


INTRODUCTION 

Test  and  evaluation  of  weapon  systems  is  accomplished 
throughout  the  acquisition  process  to  identify  and  reduce  devel¬ 
opment  risks  and  to  ensure  that  a  weapon  system  will  perform  as 
intended.  The  Department  of  Defense  (DOD)  decisionmakers  and  the 
Congress  use  the  results  to  manage  and  oversee  the  development 
and  acquisition  process.  In  fact,  DOD's  acquisition  policy 
states  that  successful  accomplishment  of  test  and  evaluation  is  a 
key  requirement  for  decisions  committing  significant  resources  to 
a  program  or  to  advance  it  from  one  acquisition  phase  to 
another.  Operational  test  and  evaluation  is  especially  important 
because  it  demonstrates  whether  a  weapon  system  will  perform  as 
intended.  The  increasing  sophistication  and  capabilities  of 
DOD's  new  weapon  systems  make  testing  even  more  critical  to 
ensure  that  expensive  weapons  achieve  the  best  performance 
poss  ible . 


This  report  discusses  DOD's  capability  to  perform  necessary 
test  and  evaluation  in  support  of  the  need  for  high  quality 


weapon  system  performance  data, 
of  test  resources  ]_/ — electronic 


The  report  focuses  on  two  types 
warfare  threat  simulators  and 


aerial  targets.  It  also  discusses  actions  necessary  to  improve 


DOD's  capability  to  support  testing. 


The  importance  of  aerial  targets  and  electronic  warfare 
threat  simulators  was  underscored  by  the  Under  Secretary  of 
Defense,  Research  and  Engineering,  when,  in  his  fiscal  year  1984 
statement  to  the  Congress  (March  2,  1983),  he  cited  these  two 
areas  as  challenges  and  recognized  that  deficiencies  exist  in 
both  threat  simulators  and  aerial  targets.  Specifically,  he 
called  for  a  coherent,  coordinated  program  to  provide  a  realistic 
threat  environment  for  testing  all  major  systems.  He  also  indi¬ 
cated  that  various  alternatives  for  a  target  to  represent  the  low 
altitude,  high  speed,  surface  ship  missile  threat  are  under 
evaluation . 


Vlest  resources  are  those  resources  necessary  to  conduct  test 
and  evaluation  of  weapon  systems  and  include  such  things  as 
data  processing  equipment,  telemetry  and  other  communications 
devices,  support  vehicles,  aerial  targets,  and  electronic 
warfare  threat  simulators. 
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TESTING  I  N  AN  ELECTRON  I C  WARFARE 

ENV  I  Rl MJ2  N  T  "  1  S  HKCOM1  N( i  MORE  I  Ml ’ORTANT 

Admiral  Thomas  Moore r  (former  Chief  of  Naval  Operations, 

U.S.  Navy)  has  said,  "ft  there  is  a  World  War  I  IT,  the  winner 
wilL  he  the  side  that  can  best  control  and  manage  the  electromag¬ 
netic  spectrum."  This  statement  emphasizes  concisely,  the 
significance  of  electronic  warfare  and  underlines  the  importance 
of  testing  weapon  systems  in  a  realistic  electronic  warfare 
env i ronmen t . 

electronic  warfare  involves  the  use  of  electromagnetic 
energy  (i.e.,  radio  waves)  to  determine,  exploit,  reduce,  or  pre¬ 
vent  the  enemy's  use  of  the  electromagnetic  spectrum.  Electronic 
warfare  affects  all  electronic  systems,  including  radars,  naviga¬ 
tion  systems,  communications,  and  guidance  systems.  In  fact, 
almost  everything  the  military  uses  can  he  severely  degraded  in 
an  electronic  warfare  environment.  A  discussion  of  electronic 
warfare  principles  is  in  Appendix  I. 

The  enemy  will  use  electronic  warfare  as  a  weapon  with  a 
specific  mission — its  doctrine  advocates  the  use  of  electronic 
warfare  as  an  element  of  combat  power.  For  this  reason,  all 
weapons  subject  to  degradation  by  electronic  warfare  should  be 
tested  in  as  realistic  and  stringent  an  environment  as  possible, 
battlefield  commanders  have  to  know  the  full  capabilities  and 
limits  of  their  weapon  systems.  Recent  experiences  in  the  Falk- 
lands  conflict  and  especially  Israel's  successes  against  Syria's 
surface-to-air  missile  systems  underscore  the  importance  of  com¬ 
plete  and  accurate  weapon  system  performance  information. 

DEVELOPMENT  OF  SUITABLE  AERIAL 
TARGETS  IS  BECOMING  MORE  DIFFICULT 


Realistic  operational  test  and  evaluation  of  weapon  systems 
requires  aerial  targets  that  test  weapon  systems  as  they  are  used 
in  combat.  Since  it  is  not  practical  to  use  actual  enemy 
aircraft  and  missiles  as  targets,  DOD  designs  and  develops 
targets  that  are  intended  to  represent  the  critical  capabilities 
and  characteristics  (speed,  altitude,  electronic  countermeasures, 
engine  heat  patterns,  and  etc.)  of  enemy  aircraft  and  missiles. 

Newer  enemy  weapons,  such  as  the  Foxbat  aircraft  and  the 
low  altitude,  supersonic  antiship  missiles,  present  difficult 
challenges  to  target  developers.  As  technology  pushes  weapon 
systems'  capabilities  to  new  heights,  the  target  developer  tries 
to  match  that  capability  with  low-cost  vehicles  that  can  be 
effectively  used  in  testing  and  training.  Target  development  is 
an  increasingly  important  task,  since  significant  weapon 
performance  limitations  may  go  undetected  unless  aerial  target 
development  keeps  pace. 
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OUR  PREVIOUS  REPORTS  HAVE  DISCLOSED 
SIGNIFICANT  TESTIN'^  ISSUES 

We  have  issued  numerous  reports  on  DOD’ s  test  and  evaluation 
process.  As  early  as  1972,  we  reported  that  most  systems  did  not 
have  adequate  test  plans  and  that  testing  was  not  accomplished  in 
a  timely  manner.  In  1975  we  reported  that  the  most  troublesome 
problem  in  test  resources  was  the  lack  of  realistic  targets. 

Other  reports  on  the  testing  function  and  our  annual  weapon  sys¬ 
tem  reports  have  also  stressed  the  need  for  improved  realism  in 
testing  and  more  representative  test  resources.  A  list  of  our 
reports  involving  testing  is  in  Appendix  II.  As  this  report 
shows,  significant  improvements  have  been  made  in  some  areas,  hut 
many  improvements  still  need  to  be  made.  (See  ch.  3.) 

THE  CONGRESS  HAS  BECOME  INCREASINGLY 
CONCERNED  ABOUT  TESTING  AND  WEAPON 
SYSTEM  PERFORMANCE 


The  Congress  has  consistently  taken  an  interest  in  the  plan¬ 
ning,  conduct,  evaluation,  and  reporting  of  test  results.  In 
1971  the  Congress  passed  Puhlic  Law  92-156,  requiring  DOD  to  pro¬ 
vide  the  Congress  with  data,  through  congressional  data  sheets, 
on  the  operational  testing  and  evaluation  of  its  weapon  systems. 
As  early  as  1969,  DOD  has  been  giving  the  Congress  quarterly 
reports  on  the  cost,  schedule,  and  performance  status  of  its 
major  weapon  systems.  Annual  hearings  on  weapon  systems  have 
continually  highlighted  weapon  performance  issues  and  the  need 
for  better  test  and  evaluation.  In  addition,  the  House  Committee 
on  Government  Operations  and  the  Senate  Committee  on  Governmental 
Affairs  plan  to  hold  hearings  during  the  summer  of  1983  on  the 
role  of  testing  and  test  resources  in  weapon  system  acquisition. 

OBJECTIVE,  SCOPE,  AND  METHODOLOGY 

Our  primary  objective  was  to  assess  DOD’ s  capability  to  test 
and  evaluate  its  weapon  systems  in  an  electronic  warfare  environ¬ 
ment  and  against  aerial  targets  that  represent  the  expected 
threat.  We  met  the  objective  through  two  means.  First,  we 
reviewed  and  updated  internal  DOD  studies  and  assessments  and  our 
prior  reports  on  testing  and  test  resources.  Second,  we  reviewed 
the  management  and  use  of  electronic  warfare  and  aerial  target 
test  resources  in  several  DOD  weapon  system  testing  programs. 

Our  fieldwork  was  conducted  from  April  through  December  1982. 

We  reviewed  prior  studies  to  determine  whether  DOD  had 
reported  any  problem  areas  concerning  the  adequacy  of  test 
resources  and  to  acquire  historical  information.  We  used  infor¬ 
mation  gained  from  the  studies  in  our  review  of  selected  weapon 
system  test  programs.  Whenever  we  used  information  from  prior 
DOD  studies  or  our  reports,  we  determined  whether  or  not  the 
condition  reported  still  existed  and  whether  any  corrective 
actions  had  been  taken. 
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To  complement  and  update  the  historical  information  provided 
by  previous  studies,  we  used  nine  DOD  weapon  system  testing  pro¬ 
grams  as  a  means  to  illustrate  the  current  state  of  DOD's  manage¬ 
ment  and  use  of  test  resources  in  general  and  electronic  warfare 
threat  simulators  and  aerial  targets  specifically.  We  selected  9 
of  88  weapons  that  either  had  been  or  would  be  tested  during 
fiscal  years  1982  and  1983.  The  test  programs  (listed  below) 
were  chosen  to  serve  as  examples  to  illustrate  the  overall 
problems  in  the  management  and  use  of  test  resources.  We  believe 
these  systems  represent  a  broad  range  in  terms  of  missions  and 
capabilities.  Time  and  other  constraints  prevented  us  from 
selecting  additional  programs.  During  our  review,  however,  we 
became  aware  of  problems  encountered  in  other  weapon  systems  and 
have  included  them  where  appropriate. 

Test  Programs  Reviewed 

Air  Force 


B-52  Offensive 
Avionics  System 
F-16  Multinational 
Staged  Improvement 
Program 

EF-111A  Tactical 
Jamming  System 

ssne 


Army 


Single  Channel 
Ground-Airborne 
Radio  System-VHF 
Tactical  Communications 
Jamming  System 
Patriot  Missile  System 


High  Speed  Anti- 
Radiation  Missile 
ALQ-126B  Defensive 
Electronic 
Countermeasures 
Set 

Phoenix  AIM-54C 


A  description  of  each  weapon  system  is  contained  in  Appendix  III. 

By  comparing  test  plans  with  test  reports  and  interviewing 
cognizant  officials,  we  determined  how  test  resource  needs  were 
identified  and  fulfilled.  We  also  determined,  through  reviewing 
documents  and  interviewing  test  officials,  how  test  resources 
were  used  and  how  test  resource  limitations  were  treated  in  test 
reports  and  other  related  documents. 

By  comparing  the  test  plans  and  reports  with  descriptions  of 
available  test  resources,  we  identified  discrepancies  between 
those  test  resources  needed  and  those  available.  The  nature  and 
extent  of  the  discrepancies,  as  well  as  the  effect  on  test  ade¬ 
quacy,  was  obtained  through  interviews  with  test  officials  and 
reviews  of  pertinent  documents. 

We  made  no  attempt  to  assess  the  effectiveness  of  either  a 
weapon  system  or  its  program  management;  nor  did  we  assess  the 
overall  adequacy  of  a  weapon's  test  program.  Consequently,  no 
conclusions  can  be  reached  regarding  the  overall  management  of  a 
particular  weapon  system  on  test  program. 

We  did  our  review  at  the  services'  operational  test  and 
evaluation  agencies,  selected  test  ranges,  operational  commands, 
weapon  system  program  offices,  the  Office  of  the  Secretary  of 


Defense  (OSD),  and  service  headquarters  offices.  Additionally 
we  contacted  selected  defense  contractors  and  related  defense 
associations.  A  complete  list  of  locations  visited  is  in 
Appendix  IV. 

Our  review  was  made  in  accordance  with  generally  accepted 
government  auditing  standards. 


CHAPTER  2 


THE  T..3T  RESOURCE  PROBLEM  TODAY 

Without  test  resources  that  adequately  duplicate  the  charac¬ 
teristics  of  the  threat,  the  true  performance  capabilities  of 
DOD's  weapon  systems  will  not  be  known,  and  significant  risks  may 
go  unexposed  until  deployment  and  actual  use.  For  example,  the 
Air  Force’s  Sparrow  air-to-air  missile  was  tested  against  aerial 
targets  that  did  not  realistically  represent  the  actual  threat. 
When  first  used  in  Vietnam,  the  Sparrow  missed  its  target  more 
often  than  it  hit  it.  The  Sparrow's  effectiveness  improved  after 
further  operational  test  and  evaluation  and  subsequent  design 
mod i f icat ions . 

DOD's  policy  1/  requires  operational  test  and  evaluation  to 
be  accomplished  in  an  environment  as  operationally  realistic  as 
possible,  using  test  resources  that  closely  resemble  the  expected 
wartime  and  peacetime  conditions.  A  realistic  environment  is 
especially  necessary  when  testing  weapons  that  may  be  used 
against  Soviet  aircraft  and  missiles  and  when  testing  in  an  elec¬ 
tronic  warfare  environment. 

The  development  of  realistic  electronic  warfare  threat  simu¬ 
lators  and  aerial  targets  has  not  kept  pace  with  the  increasing 
complexity  and  performance  levels  of  weapon  systems  used  by  our 
adversaries.  As  a  result,  DOD's  weapon  systems  are  being  tested 
in  environments  that  do  not  adequately  represent  the  conditions 
in  which  the  weapons  are  expected  to  perform.  Important  and 
costly  weapons,  such  as  the  Navy's  CG-47  Aegis  cruiser,  the 
improved  Phoenix  air-to-air  missile,  the  Army's  Patriot  missile 
system,  and  the  Air  Force's  B-52  offensive  avionics  system,  will 
be  deployed  without  assurance  that  they  will  meet  performance 
expectations . 

PRACTICAL  CONSTRAINTS  ON  TESTING 

A  totally  realistic  operational  environment  is  difficult  to 
achieve  and,  therefore,  operational  testing  cannot  always  be  as 
realistic  as  it  should.  Practical  limitations,  such  as  range 
size  or  nearness  to  population  centers,  may  limit  the  use  of  test 
resources  and  the  weapons  being  tested.  High  powered  jammers, 
for  example,  may  not  be  used  near  other  communications  equip¬ 
ment.  In  addition,  under  certain  circumstances,  long-range  mis¬ 
siles  or  targets  cannot  be  flown  over  populated  areas.  These 
considerations  and  others  will  always  be  limiting  factors  on 
realistic  testing,  but  improvements  can  still  be  made  in  the 
management  and  development  of  test  resources  that  will  lead  to 
more  realistic  testing  and  better  system  performance. 


1  'DOD  Directive  5000.3,  Test  and  Evaluation,  December  26,  1979. 


LIMITED  ELECTRONIC  WARFARE  TEST  CAPABILITIES 


The  realism  of  the  test  environment  directly  affects  the 
validity  of  test  results.  Typically,  however,  most  environments 
at  test  ranges  and  facilities  do  not  adequately  approximate  the 
real  threat  because  electronic  warfare  threat  simulators  are  not 
available  in  sufficient  quality,  numbers,  or  types. 

A  1982  study  sponsored  by  OSD  reported  that  the  number  of 
simulators  needed  clearly  exceeds  anticipated  resources:  Of  84 
test  programs  scheduled  between  fiscal  years  1982  and  1987,  at 
least  60  may  face  threat  simulator  shortfalls. 

The  United  States  inventory  of  threat  simulators  does  not 
cover  many  of  the  enemy' s  weapon  systems  that  our  weapons  may 
face.  As  the  following  chart  illustrates,  some  threat  systems 
have  no  simulator  coverage  at  all. 


Coverage  of  Potential  Threat  Systems 
With  Existing  United  states  Simulator  Resources 

(as  of  1981) 


Percent 

Type  of  threat  simulated 

Early  warning/ground  controlled  intercept  radars  34 

Surface-to-air  missile  radars  61 

Antiaircraft  artillery  radars  60 

Airborne  intercept  radars  27 

Infrared  systems  50 

Ground-based  jammers  14 

Shipborne  jammers  0 

Airborne  jammers  0 


Source:  Derived  from  a  comparison  of  data  contained  in  the 
Threat  Simulator  Master  Plan,  OSD/Director,  Defense 
Test  and  Evaluation,  February  1982. 

Other  service  studies  have  identified  additional  problems 
with  threat  simulators  and  our  audit  work  showed  that  they  con¬ 
tinue  to  exist.  Some  of  these  problems  are  discussed  below. 


A  1973  Naval  Audit  Service  report  on  naval  electronic  war¬ 
fare  capabilities  revealed  major  deficiencies  in  the  Navy's 
ability  to  test  and  eviluate  its  weapon  systems  in  an  electronic 
warfare  environment.  .he  report  identified  lack  of  visibility  at 
higher  management  levels,  inadequate  funding  and  planning,  and 
fragmented  management  as  the  basic  causes.  Further,  the  Navy's 
current  Electronic  Warfare  Master  Plan,  prepared  in  1981,  recog¬ 
nized  several  problems.  The  plan  indicated  an  overall  lack  of 
simulation  equipment  to  generate  an  adequate  operational  environ¬ 
ment  for  either  testing  or  training. 

The  Army,  in  its  Mission  Element  Need  Statement  for  the  Army 
Development  and  Acquisition  of  Threat  Simulators  program, 
stated  that  a  satisfactory  realistic  threat  environment  for  test¬ 
ing  cannot  be  provided.  A  1982  study  sponsored  by  OSD  said  that 
most  range  environments  cannot  approximate  the  real  threat 
because  the  requisite  number  and  types  of  simulators  are  not 
available.  The  report  attributed  the  nonavailability  of  threat 
simulators  to  a  variety  of  circumstances,  including  insufficient 
funding,  intelligence  gaps,  and  incomplete  correlation  of  program 
requirements  with  acquisition  plans. 

The  Air  Force,  in  a  1982  evaluation  of  electronic  warfare 
test  capabilities,  concluded  that  present  threat  simulator  capa¬ 
bilities  cannot  support  known  and  projected  test  requirements. 
Further,  the  differences  between  existing  and  required  test  capa¬ 
bilities  were  reported  to  be  increasing,  because  of 

— a  rapid  increase  in  the  number  of  threat  systems, 

— the  time  required  to  obtain  sufficient  intelligence  data 
on  the  threat  systems, 

— the  lead  times  necessary  to  build  or  modify  test  systems 
and  ranges,  and 

—  threat  simulator  budget  limitations. 

The  limitations  identified  in  these  studies  have  led  to  a 
number  of  significant  shortcomings  in  the  testing  of  several 
weapon  systems.  The  Navy,  for  example,  cannot  fully  assess  the 
performance  of  several  of  its  systems  in  an  operational  environ¬ 
ment  because  of  the  lack  of  suitable  simulators.  Some  simulators 
have  not  been  developed  while  others  do  not  adequately  represent 
the  threat  they  are  designed  to  simulate. 

The  Air  Force's  new  offensive  avionics  system  for  the  B-52 
experienced  similar  shortcomings  in  threat  simulator 
availability.  Not  all  of  the  test  resources  (simulators  of  enemy 
radar  jammers)  necessary  to  obtain  sufficient  assurance  of 
operational  effectiveness  were  available.  The  test  manager  was 
not  satisfied  that  the  system  could  successfully  counter  the 


threat  and  recommended  further  testing  with  analytical  simula¬ 
tions  and  computer  models.  According  to  the  test  manager, 
however,  simulations  and  computer  models  to  support  recommen¬ 
dations  contained  in  the  test  report  are  not  available. 
Consequently,  the  performance  of  the  system  in  its  operational 
environment  cannot  be  adequately  assessed. 

Discussions  with  various  service  officials  have  surfaced 
other  contributing  causes  to  the  test  resource  problems.  Some 
suggested  that  many  military  commanders  do  not  fully  appreciate 
the  intricacies  of  electronic  warfare  or  its  potentially 
devastating  effects.  The  recent  emphasis  on  shortening  weapon 
system  acquisition  time  has  also  been  seen  by  others  as 
conflicting  with  thorough  testing. 

LIMITED  AERIAL  TARGET  CAPABILITIES 


Several  major  weapon  systems — including  the  Advanced  Medium 
Range  Air-to-Air  Missile  (AMRAAM),  the  improved  Phoenix  missile, 
and  the  Patriot  Air  Defense  Missile  System — will  be  fielded 
without  complete  testing  because  existing  aerial  targets  cannot 
adequately  test  the  systems'  capabilities.  An  urgent  need  exists 
for  targets  that  can  realistically  duplicate  the  high  altitude, 
high  speed  threat  aircraft  and  antiship  cruise  missiles,  and  the 
low  altitude,  high  speed  antiship  cruise  missiles.  The  Air 
Force's  planned  Firebolt  target  may  eventually  satisfy  the  high 
speed,  high  altitude  requirements,  but  limitations  in  other 
areas — its  small  size,  its  nonrealistic  radar  and  infrared 
signatures,  and  its  lack  of  an  electronic  countermeasures 
capability — will  significantly  reduce  its  usefulness. 

The  Firebolt  is  scheduled  to  enter  production  in  1985,  after 
the  testing  of  several  major  weapon  systems  has  been  completed. 
The  Army,  for  example,  needs  the  Firebolt  target  to  evaluate  the 
Patriot  against  a  high  speed/high  altitude  threat.  However,  the 
Firebolt  will  not  be  produced  in  time  for  testing  with  the 
Patriot  system.  In  addition,  the  Air  Force's  AMRAAM  will  not  be 
realistically  tested  against  the  high  altitude,  high  speed  threat 
before  production  because  the  Firebolt  will  not  be  ready  in 
time.  Consequently,  these  weapons  may  be  approved  for  production 
with  unknown  performance  capabilities  or  the  production  decision 
could  be  delayed  because  of  insufficient  performance  data. 

No  existing  target  can  reasonably  duplicate  the  threat  posed 
by  the  sea-skimming  supersonic  cruise  missile.  The  Navy's 
Firebrand  was  to  have  filled  this  void  but  it  was  cancelled  by 
the  Navy  because  it  could  not  adequately  represent  the  threat  and 
was  not  cost  effective.  As  a  replacement,  the  Navy  is  planning 
to  modify  an  existing  target  (the  Vandal)  as  a  short-term 
solution.  The  Vandal,  however,  will  not  adequately  duplicate  the 
sea-skimming  cruise  missile.  For  the  long  term,  the  Navy  plans 
to  develop  a  supersonic  low-altitude  target.  The  Vandal  is 


scheduled  to  be  available  shortly,  but  a  supersonic  low-altitude 
target  will  not  be  available  until  much  later.  As  a  result,  the 
U.S.S.  Ticonderoga  equipped  with  the  Aegis  fleet  air  defense 
system  will  be  deployed  without  demonstrating  an  ability  to 
defeat  a  supersonic  sea-skimming  cruise  missile. 

Delays  in  developing  the  Firebolt  and  the  cancellation  of 
the  Firebrand  have  forced  the  services  to  rely  on  segmented  test¬ 
ing  using  a  "family  of  targets"  approach.  The  current  "family," 
however,  includes  the  same  targets  with  the  same  major  limita¬ 
tions  that  the  Firebolt  and  the  Firebrand  were  to  overcome. 

An  April  1981  study  sponsored  by  OSD  stated  that  no  existing 
aerial  target  can  satisfy  all  minimum  testing  needs  in  the  areas 
of  speed,  altitude,  maneuverability,  endurance,  scoring,  and 
radar  and  infrared  signatures.  Air  Force  officials  responsible 
for  developing  aerial  targets  stated  that  these  limitations  still 
exist,  are  restricting  testing,  and  are  critical  to  successful 
test  and  evaluation. 

The  Air  Force  Inspector  General,  in  a  1979  report,  concluded 
that  aerial  targets  had  low  priority  and  visibility  at  higher 
management  levels  and  sufficient  funding  had  not  been  programmed, 
although  the  using  commands  demonstrated  a  need  for  the  targets. 
These  factors  still  exist  and  significantly  contributed  to  delay¬ 
ing  deployment  of  the  Air  Force's  Firebolt  target  system.  Orig¬ 
inally  planned  for  1981,  deployment  is  now  scheduled  for  1985. 

The  Inspector  General's  report  also  surfaced  an  important 
issue  that  underlies  many  of  the  problems  affecting  all  test 
resources.  The  report  pointed  out  that  the  Air  force  aerial 
targets  program  is  formally  managed  as  a  weapon  system  and  con¬ 
tends  for  attention  and  first  level  trade-offs  with  weapon  sys¬ 
tems  such  as  the  F-15,  air  intercept  missiles,  and  the  Low- 
Altitude  Airfield  Attack  System.  Conversely,  target  users  view 
aerial  targets  as  test  and  evaluation  and  training  devices.  The 
report  stated  that  aerial  targets  are  of  little  use  in  and  to 
themselves  and,  therefore,  should  neither  be  considered  nor  form¬ 
ally  managed  as  weapon  systems. 

CONCLUSIONS 

Although  the  services  have  recognized  the  inability  of 
existing  electronic  warfare  threat  simulators  and  aerial  targets 
to  adequately  meet  present  and  future  test  requirements,  their 
efforts  to  resolve  the  problems  have  not  been  successful.  Threat 
simulators  used  in  testing  weapons  in  electronic  warfare 
environments  are  limited  in  both  quality  and  quantity.  In  a 
number  of  cases,  no  simulators  exist  or  are  planned  for  threats 
.that  have  been  in  existence  for  years.  Furthermore,  the  services 
still  lack  sufficient  aerial  targets  to  assess  the  performance  of 
weapon  systems  against  the  enemy's  increasingly  sophisticated 
systems . 


Numerous  studies  demonstrate  that  the  problems  are  well 
known  and,  more  importantly,  persistent.  Several  of  the  studies 
suggest  the  problems  are,  at  least  partially,  due  to  a  lack  of 
agreement  within  DOD  on  the  importance  of  threat  simulators  and 
aerial  targets.  The  Air  Force  has  pointed  to  the  management  of 
targets  with  weapons  and  the  resultant  trade-offs  between  F-15s 
and  targets.  Other  studies  have  suggested  that  military  command¬ 
ers  do  not  fully  appreciate  the  devastating  effects  electronic 
warfare  can  have  on  a  weapon's  effectiveness.  Still  others  have 
pointed  to  the  inherent  conflict  between  thorough  testing  and 
DOD's  current  emphasis  to  accelerate  weapon  system  acquisition 
time.  Consequently,  electronic  warfare  testing  and  aerial 
targets  have  received  limited  resources  and  have  not  kept  pace 
with  the  weapon  systems  that  need  them  for  thorough  testing. 


In  Chapter 
test  capability 
greatly  improve 
limitations  are 


3,  the  underlying  causes  of  limitations  in  DOD's 
are  examined.  Solutions  which  we  believe  will 
the  service's  efforts  to  eliminate  these 
also  provided. 
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CHAPTER  3 


RESOLVING  TEST  RESOURCE  PROBLEMS  FOR  THE  FUTURE 


Designing  and  developing  suitable  test  resources  to  realis¬ 
tically  simulate  enemy  weapons  is  very  difficult,  and  test 
resource  developers  must  overcome  many  obstacles.  In  some  areas, 
such  as  the  acquisition  of  precise  miss-distance  indicators  for 
aerial  targets,  technology  has  delayed  development.  In  other 
instances,  the  need  to  protect  intelligence  data  and  sources  has 
restricted  the  development  and  use  of  certain  enemy  electronic 
warfare  equipment  simulators.  Even  when  test  resources  are 
developed,  other  factors  may  limit  their  use.  High  powered  jam¬ 
mers,  for  example,  cannot  be  used  near  civilian  communications 
equipment . 

Many  reasons  delay  or,  in  rare  instances,  prevent  the  timely 
development,  acquisition,  and  use  of  threat  simulators  and  aerial 
targets.  Four  areas,  however,  are  so  pervasive  they  affect 
almost  every  threat  simulator  and  target  development  program. 
These  areas,  basic  to  any  well  organized  endeavor,  are 

— planning , 

— organiz  ation , 

— management  emphasis,  and 

—  intelligence  support. 

Successful  accomplishment  of  these  functions,  fundamental  to 
good  management,  will  remove  some  of  the  most  serious  obstacles 
to  satisfactory  threat  simulator  and  aerial  target  development. 
Until  development  programs  are  properly  planned,  better 
organized,  given  adequate  management  emphasis,  and  obtain 
sufficient  intelligence  support,  it  is  a  virtual  certainty  that 
needed  resources  will  not  be  available  when  weapons  are  tested. 

In  addition,  we  believe  the  problems  in  planning  and  management 
emphasis  are  systemic  and  adversely  affect  testing  and  test 
resources  in  general. 

As  Chapter  2  demonstrated,  there  are  severe  shortages  in  the 
number  and  types  of  electronic  warfare  threat  simulators  and  in 
realistic  aerial  targets.  As  a  result,  DOD  is  fielding  weapon 
systems  without  sufficient  knowledge  of  their  ability  to  survive 
or  function  in  combat.  Field  commanders  are  operating  weapons 
with  unknown,  perhaps  dangerous,  limitations. 
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rurrer't  ooij  policy  and  implementing  service  regulations 
require  dr  '  =; p r ,  system  developer  and  the  operational  test 
agency  -  •  •-  i.  i-_- 1  j i  Led  test  planning  before  starting  the  demon¬ 
strati  or.  >,  i  i  ;  i  ;n  phase  (Milestone  I)  of  the  acquisition 
process.  1  nose  early  test  plans,  called  Test  and  Evaluation  Mas¬ 
ter  Plans  "id-M-sl,  are  to  contain  all  of  the  basic  elements  nec¬ 
essary  1  .  •  id«.-nti  f  ication,  development,  and  acquisition  of 

test  run  r  •  ,  require  OSD  approval.  Specifically,  the  test 

plans  arc  t  •  c'c'c  a  i  n 

—  a  description  of  the  weapon  system  to  be  tested; 

—  a  discussion  of  the  critical  test  and  evaluation 

issues ; 

--an  outline  of  the  planned  testing;  and 

--specific  identification  of  key  test  resources,  including 

equipment ,  personnel,  and  funds. 

TEMPs-- an  under  used  keystone 
for  early  te st  v 1 a n n i n g 

The  lmpne-t -nice  of  early  test  planning  was  emphasized  by  the 
Under  Sec te tar. ,  Defense  Research  and  Engineering,  in  an 
August  i ,  L '•) fi  : ,  memorandum  to  the  services.  He  said  that  proper 
planning  is  ■  -ssent i a  l  to  improving  the  acquisition  process  and 
that  p r  i > .•  >•  .i  f  y  j *  *-.en i on  must  be  given  to  incorporating  test 
requirements  into  established  planning  documents  such  as  the 
TEMP.  He  further  stated  he  intends  to  use  TEMPs  to  "clearly 
identify  *  ost  uirce  requirements." 


It  ;  .  *,  :  PEMP  can  serve  as  the  basic  test  document 
to  do:  i  ,  ■  •  - ; i J  test  resource  needs.  However,  TEMPs 
have  r>  >t  ■  their  potential.  We  found  cases 
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where  TEMPs  were  either  too  late,  incomplete,  or  both,  to  be 
satisfactorily  used  by  decisionmakers. 

For  example,  the  TEMP  on  the  EF-lllA  aircraft  program  did 
not,  as  required  by  current  DOD  guidance, 

—  specify  the  kinds  of  threat  simulators  needed, 

—  identify  steps  necessary  to  acquire  needed  simulators, 
or 

—  identify  expected  test  limitations  and  the  adverse  effects 
of  those  limitations. 

The  EF-lllA  test  program  had  significant  limitations  that 
adversely  affected  the  scope  and  completeness  of  the  testing. 

The  operational  tests  were  limited  to  a  single  EF-lllA,  escorting 
four  other  aircraft,  striking  a  target  complex  defended  by  only  a 
handful  of  surveillance  and  tracking  radars.  Therefore, 
electronic  signal  density  was  significantly  less  than  the 
expected  combat  environment  and  reduced  the  quality  of  the  test 
data.  In  addition,  the  evaluation  of  the  EF-lllA' s  other  support 
roles  were  limited  by  a  scarcity  of  test  resources.  According  to 
a  1982  Air  Force  study  of  electronic  combat  test  capabilities, 
the  shortages  are  continuing  and  will  adversely  affect  the  test 
and  evaluation  of  the  EF-lllA' s  upgrade  program.  These  testing 
limitations  were  known  before  testing  and  should  have  been 
surfaced  in  the  early  TEMPs  so  decisionmakers  could  assess  the 
risks  of  incomplete  testing. 

The  lack  of  early  planning  affected  the  EF-lllA' s  testing  in 
other  ways.  The  Air  Force  Test  and  Evaluation  Center  presented 
its  test  resource  requirements  to  a  test  facility  only  4  months 
before  testing  was  scheduled  to  begin.  The  facility  stated  it 
would  need  at  least  7  months  to  acquire  the  necessary  resources. 
Consequently,  only  limited  testing  was  accomplished. 

The  B-52  offensive  avionics  program  is  another  example  that 
illustrates  the  problems  of  inadequate  test  planning  and 
insufficient  test  resources.  The  program's  TEMP  did  not  specify 
the  types  of  test  resources  needed,  whether  resources  were 
available,  or  the  effect  of  not  having  adequate  resources.  The 
B-52  could  not  be  adequately  tested  against  recognized  threats 
because  threat  simulators  were  not  available  in  sufficient 
quantity.  Thus,  the  B-52's  capability  is  not  fully  known. 

Again,  these  limitations  should  have  been  surfaced  before  testing 
began . 

We  attempted  to  quantify  the  problem  of  late  and  incomplete 
TEMPs  but,  because  the  necessary  records  were  not  available  at 
DOD,  were  unable  to  accomplish  this  task.  Other  evidence, 
however,  suggests  the  problem  is  widespread.  Several  service  and 
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OSD  officials  responsible  for  reviewing  and  approving  TEMPs 
complained  of  late  and  incomplete  TEMPs.  This  view  is 
corroborated  by  a  1981  study  for  the  Defense  Systems  Management 
College.  In  their  review  of  six  weapon  system  test  programs, 
four  did  not  have  TEMPs  early  enough  to  effect  the  adequacy  of 
test  and  evaluation  decisions.  The  problems  are  continuing;  the 
Tomahawk  cruise  missile  is  in  production  but  still  does  not  have 
an  OSD  approved  TEMP. 

The  Airborne  Self-Protection  Jammer  program,  on  the  other 
hand,  is  an  example  of  where  a  TEMP  has  been  properly  prepared. 

It  contains  a  comprehensive  list  of  necessary  test  resources, 
including  some  that  are  not  in  DOD's  inventory.  These  resources 
are  identified  as  being  critical  to  determining  the  combat  effec¬ 
tiveness  of  the  jammer. 

We  do  not  believe  that  by  just  enforcing  the  TEMP  require¬ 
ments,  adequate  test  resources  will  be  available.  Indeed,  we 
found  examples  where  specific  test  resources  were  identified  as 
not  available  and  the  resources  were  still  not  available  when 
needed.  The  High  Speed  Anti-Radiation  and  Phoenix  missiles  had 
TEMPs  that  clearly  identified  specific  targets  but,  because  of 
other  problems,  the  targets  were  never  developed.  However,  with¬ 
out  adequate  planning  test  resource  needs  cannot  be  identified 
and  acquisition  programs  cannot  be  started. 

ORGANIZATIONAL  STRUCTURES  SHOULD 
NOT  CREATE  DEVELOPMENT  BARRIERS 


The  Navy's  and  the  Air  Force's  current  organizational 
structures  for  developing  threat  simulators  and  aerial  targets 
are  fragmented  with  no  clear  assignment  of  management 
responsibility  among  the  many  organizations  involved.  in  some 
cases,  the  structure  allows  direct  competition  between  weapon 
systems  and  the  test  resources  needed  to  evaluate  them.  To  allow 
the  same  manager  to  control  a  weapon  development  program  and  the 
test  resources  used  to  evaluate  the  success  of  that  weapon 
creates,  in  our  opinion,  a  conflict  of  interest. 

Navy  threat  simulator  management — 
fragmented  with  conflicting  interests 

Fragmented  management 


The  Navy' s  management  of  threat  simulator  development  is 
fragmented,  with  no  single  office  in  charge.  This  fragmentation 
impedes  the  Navy's  efforts  to  acquire  threat  simulators  because 
it  diffuses  responsibility  and  makes  coordination  and  overall 
program  direction  difficult. 

Problems  with  the  Navy's  current  management  of  threat 
simulators  are  recognized  by  the  Navy  but  little  action  has  been 
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taken  to  correct  them.  A  1978  Naval  Audit  Service  report  stated 
that  due  to  the  lack  of  adequate  planning  and  funding,  as  well  as 
fragmented  management,  the  Electronic  Warfare  Threat  Environment 
Simulation  Facility  at  the  Naval  Weapons  Center  " .  .  .does  not 
have  the  prerequisite  capabilities  to  perform  its  mission." 

(This  facility  is  the  Navy's  primary  electronic  warfare  test 
range.)  A  1980  study  by  the  Naval  Air  Systems  Command  identified 
the  fundamental  issue  underlying  many  of  the  study's  findings  as 
fragmentation  of  management  at  both  the  Chief  of  Naval  Operations 
and  lowei.  command  levels,  resulting  in  insufficient  coordination, 
inadequate  or  duplicate  requirements,  inefficient  support 
planning,  and  a  lack  of  cost  effectiveness  in  the  acquisition, 
support,  and  use  of  threat  simulators. 

The  Navy  Electronic  Warfare  Master  Plan,  issued  in  1981, 
also  highlighted  fragmented  management.  It  identified  11  types 
of  Navy  organisations,  ranging  from  the  operational  commands  to 
the  test  and  evaluation  community,  that  have  electronic  warfare 
responsibilities.  The  report  stated  that  the  various  organiza¬ 
tions  have  overlapping  structures  and  approach  electronic  warfare 
in  terms  of  their  own  environment,  motivations,  and  limitations. 
Within  the  office  of  the  Chief  of  Naval  Operations,  the  Elec¬ 
tronics  Warfare  Division  is  responsible  for  coordinating  the 
acquisition  of  threat  simulators,  but  has  no  funding  or 
management  authority.  As  the  1981  Electronic  Warfare  Master  Plan 
points  out,  however,  mere  coordination,  without  resource  control, 
is  not  enough.  In  spite  of  the  previous  studies,  fragmented 
management  of  electronic  warfare  test  resources  continues  within 
the  Navy. 

Conflicting  interests 


In  at  least  one  case,  the  Navy's  organizational  structure 
may  have  contributed  to  a  lack  of  test  capability  at  the  Navy's 
primary  electronic  warfare  test  range. 

The  Naval  Air  System  Command's  program  manager  for  airborne 
electronic  warfare  devices  is  also  responsible  for  planning, 
budgeting,  and  acquiring  simulators  at  the  Navy's  Electronic 
Warfare  Threat  Environment  Simulation  Facility.  Thus,  the  same 
person  is  in  charge  of  both  developing  electronic  warfare  equip¬ 
ment  and  controlling  the  development  of  the  threat  simulators 
necessary  to  test  that  equipment.  Placing  responsibility  for 
both  developing  and  testing  weapon  systems  with  one  person  or 
office  results  in  conflicting  interests  and  may  result  in  a  sit¬ 
uation  where  several  weapons  could  be  funded  at  the  expense  of 
the  equipment  needed  to  determine  the  combat  worth  of  those 
weapons.  Although  no  direct  connection  can  be  made,  the  1978 
Naval  Audit  Service  Report  states  the  electronic  warfare  test 
facility  does  not  have  the  resources  to  accomplish  its  mission. 
Our  fieldwork  also  showed  that  the  test  facility  continues  to 
experience  significant  shortfalls  in  acquiring  threat  simula- 
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lots.  We  also  note  that  of  the  24  Navy  test  programs  included  in 
the  1982  Threat  Simulator  Master  Plan,  all  will  face  threat 
simulator  shortfalls. 

Air  Force  test  resource  management — 
also  fragmented  with  conflicting  interests 


Air  Force  threat  simulator  and  target  development  programs 
are  also  impeded  by  current  organizational  structures.  No  single 
focal  point  or  "in-charge"  office  for  threat  simulators  exists 
and  the  aerial  target  program  is  in  direct  competition  with  Air 
Force  missiles  for  funds,  priority,  and  personnel. 

Threat  simulators 


A  1982  Air  Force  Systems  Command  study  on  electronic  warfare 
test  capability  concluded  that  present  threat  simulator  capa¬ 
bilities  cannot  support  known  and  potential  test  requirements  and 
that  the  gap  is  widening.  It  cited  fragmented  management  and 
budget  approval  processes  as  two  of  the  major  causes.  For 
example,  in  studying  the  Air  Force  Program  Objective  Memorandum 
review  process,  the  Air  Force  found  that  program  decision  pack¬ 
ages  for  threat  simulator  development  are  fragmented — no  single 
program  element  includes  all  threat  simulator  requirements,  and 
no  single  Air  Staff  review  board  reviews  all  test  resource  pro¬ 
gram  elements.  Consequently,  no  single  organization  or  office 
controls  the  approval  of  threat  simulator  requirements.  In 
addition,  proposed  threat  simulators  often  must  compete  with  pro¬ 
posed  electronic  warfare  equipment  within  a  single  program  ele¬ 
ment.  This  arrangement  results  in  test  equipment  competing  for 
funds  with  the  weapon  system  it  will  be  used  against,  similar  to 
the  situation  in  the  Navy. 

According  to  an  Air  Force  threat  simulator  developer,  the 
fragmentation  of  management  and  budget  approval  means  there  is  no 
structured  way  to  identify  and  coordinate  threat  simulator 
requirements.  He  added  that  the  absence  of  a  sound  organi¬ 
zational  structure  has  contributed  to  the  current  shortage  of 
threat  simulators. 

Aerial  targets 


The  Air  Force's  aerial  targets  program  office  is  located  in 
the  Air  Force  Systems  Command's  Armament  Division  at  Eglin  Air 
Force  Base,  Florida.  The  program  office  is  under  the  Deputy 
Commander  for  Counter-Air  Systems  who  is  also  responsible  for 
developing  weapons  such  as  the  AMRAAM.  This  organizational 
arrangement  places  aerial  targets  in  direct  conflict  with  the 
weapons  they  will  be  used  against  in  testing  similar  to  the 
Navy's  organizational  arrangement  discussed  earlier. 
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The  previously  mentioned  1979  Air  Force  inspector  General 
report  noted  that  targets  were  being  formally  managed  as  if  they 
were  weapon  systems.  As  a  result,  the  targets  were  susceptible 
to  trade-offs  with  weapons  such  as  the  F-15  and  air  intercept 
missiles.  The  report  also  said  this  competitive  structure  does 
not  allow  a  sufficient  evaluation  of  the  need  for  targets.  A 
recommendation  was  made  to  transfer  aerial  targets  from  the 
counter-air  mission  area  to  the  defense-wide  management  and 
support/ test  and  evaluation  support  mission  area,  to  more 
appropriately  compete  with  other  test  and  evaluation  resources 
and  not  with  weapon  systems. 

As  recommended,  aerial  targets  were  transferred,  but  in  June 
1982,  they  were  transferred  back  to  the  counter-air  mission  area 
to  compete  with  higher  priority  weapon  systems.  In  August  1982, 
the  aerial  targets  program  office  lost  21  personnel  to  the  AMRAAM 
program  office  because  AMRAAM  is  a  higher  priority  program.  This 
action  is  the  result  of  competition  the  Air  Force  Inspector 
General  warned  against  in  1979. 

Army  management — single  manager  concept 

The  Army  Development  and  Acquisition  of  Threat  Simulators 
program  was  initiated  in  1972  to  develop  threat  simulators  for 
the  Army.  In  1981,  recognizing  that  significant  deficiencies  in 
the  Army's  capability  to  test  in  a  realistic  battlefield  environ¬ 
ment  existed,  several  changes  were  made  in  the  threat  simulator 
program  that  are  intended  to  improve  its  ability  to  develop  and 
deploy  realistic  threat  simulators.  The  changes 

— tied  threat  simulator  requirements  to  threat  assessments 
rather  than  test  plans  and 

— created  a  separate  program  element  for  threat 
simulators,  reducing  internal  competition. 

By  generating  threat  simulator  requirements  from  threat 
assessments  instead  of  weapon  system  test  plans,  the  Army  does 
not  rely  on  weapon  system  developers  to  provide  threat  simulator 
requirements.  Further,  creation  of  a  separate  program  eliminates 
direct  competition  between  threat  simulators  and  weapon  systems 
and  allows  trade-off  decisions  to  be  made  at  Army  Headquarters 
level . 

Although  the  Army's  new  organizational  structure  has  not  had 
time  to  prove  itself,  we  believe  it  should  result  in  a  wider 
variety  and  better  quality  threat  simulators  that  more  accurately 
simulate  a  given  threat.  We  further  believe  centralized  control 
should  result  in  decisions  based  on  a  broader  view  of  force 
requirements  and  eliminate  the  inherent  conflicting  interests 
when  weapon  system  developers  choose  between  a  weapon  system  and 
threat  simulators. 
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MANAGEMENT  EMPHASIS  OF  TESTING 
AND  TEST  RESOURCES  IS  IMPORTANT 


Although  test  resources  are  critical  to  the  test  and 
evaluation  process,  the  amount  of  management  emphasis  received 
has  not  been  commensurate  with  their  importance.  All  too  often, 
the  test  resource  budget  is  cut  when  budgets  are  pared. 

Priority  and  funding  decisions  for  test  resources  begin  in 
the  early  planning  stages  of  weapon  system  test  programs.  At 
this  point,  needed  test  resources  should  be  identified,  their 
importance  to  the  overall  test  and  evaluation  capability 
determined,  and  actions  initiated  to  ensure  that  resources  will 
be  ready  when  needed.  As  we  have  seen,  however,  early  planning 
does  not  always  identify  the  necessary  test  resources  or  assess 
their  importance  to  testing.  Consequently,  the  resources  do  not 
receive  the  management  support  necessary  to  ensure  their  timely 
development. 

Even  after  development  programs  are  begun,  organizational 
structures  make  test  resources  easy  candidates  for  budget 
reductions  and  schedule  delays.  The  Air  Force's  Firebolt  aerial 
target  program  is  a  good  example.  The  target  program  is  under 
the  same  management  as  weapons  such  as  the  AMRAAM  and  is 
constantly  competing  for  funds  and  priority  with  these  weapons. 
Although  the  Firebolt  is  needed  to  support  the  testing  programs 
of  many  weapons,  the  Air  Force  deferred  all  fiscal  year  1984 
production  funding  and  did  not  budget  production  funds  for  fiscal 
year  1985.  Instead,  the  priority  and  funding  were  given  to  the 
missile  program.  At  the  insistence  of  OSD,  the  Air  Force  has 
restored  the  fiscal  year  1985  production  funding. 

Electronic  warfare  threat  simulators  have  similar  problems. 
According  to  a  1982  Air  Force  study,  test  capability  improvements 
are  not  available  when  needed,  primarily  because  of  insufficient 
funding.  The  study  cited  many  previous  studies  that  identified 
threat  simulator  problems  and  made  recommendations  for 
improvements  which  were  under  consideration  at  the  time  our  audit 
work  was  completed.  A  major  finding  of  the  current  study, 
however,  was  that  past  studies'  recommendations  were  not 
implemented.  We  believe  a  lack  of  adequate  priority  prevented 
implementation  of  the  recommendations. 

According  to  Navy  test  representatives,  many  of  the  Navy's 
threat  simulator  deficiencies  have  also  been  known  for  years,  yet 
the  Navy  has  taken  few  actions  to  correct  the  deficiencies.  For 
example,  a  review  by  the  Naval  Audit  Service  in  1978  found  that 
the  Navy  could  only  conduct  limited  test  and  evaluation  of  its 
airborne  electronic  warfare  equipment  because  it  lacked  adequate 
test  resources.  The  audit  report  recommended  several  actions 
to  upgrade  test  resource  limitations,  and  the  Chief  of  Naval 
Operations  agreed.  Almost  5  years  later,  however,  the  Navy  has 
yet  to  approve  and  implement  a  plan  to  correct  the  deficiencies. 
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Several  top  service  officials  also  share  our  concern  over 
the  general  lack  of  support  given  testing  and  test  resources. 

The  Director  of  the  Navy's  Office  of  Research,  Development,  Test, 
and  Evaluation,  for  example,  has  stated  that  weapon  developers 
generally  do  not  want  to  fund  test  resources  and  will  use  funds 
that  were  earmarked  for  test  resources  for  other  things.  He  said 
the  attitude  of  nonsupport  for  testing  starts  at  the  OSD  level. 
The  philosophy  seems  to  be  "testing  is  delaying  things — design  it 
and  produce  it--reduce  testing."  Further,  the  Navy's  former 
Director  of  Electronic  Warfare  told  us  that  he  believes  the 
services  should  field  weapon  systems  as  they  are  built  and  fix 
them  later  through  preplanned  product  improvement  programs.  This 
strategy,  however,  was  designed  to  take  advantage  of  advancing 
technology  and  not  as  a  substitute  for  incomplete  testing  before 
production  and  deployment. 

INTELLIGENCE  SUPPORT  IS  PIVOTAL  TO 
EFFECTIVE  TEST  AND  EVALUATION 

Since  electronic  warfare  threat  simulators  are  designed  to 
duplicate  the  enemy's  weapon  systems,  their  designs  must  be  based 
on  accurate  intelligence  information.  A  threat  simulator  based 
on  inaccurate  or  incomplete  intelligence  information  is  virtually 
worthless  and  any  evaluations  based  on  such  a  simulator  lack 
credibility.  Yet  the  services  are  using  simulators  developed 
with  inaccurate  and  incomplete  intelligence  data. 

A  1981  Defense  Audit  Service  Report  states: 

"...  there  was  no  assurance  that  the  operational  tests 
of  Air  Force  and  Navy  weapons  systems  were  conducted  in  a 
realistic  threat  environment  or  that  the  results  of  the 
tests  represented  true  system  effectiveness  against 
enemy  threats  expected  to  be  encountered  in  combat." 

The  report  cited  a  lack  of  intelligence  support  in  the  Air  Force 
and  the  Navy  as  the  primary  cause.  Our  review  found  many  of  the 
same  problems.  Officials  at  each  of  the  test  ranges  we  visited 
complained  of  known  inaccurate  and  unvalidated  threat  simulators 
on  their  ranges. 

For  example,  an  Air  Force  test  facility  commander  asked  the 
Air  Force's  Foreign  Technology  Division  to  provide  intelligence 
data  to  support  the  design  of  a  laboratory  threat  simulator  being 
developed  by  the  test  facility.  The  data  was  provided  by  the 
Foreign  Technology  Division  and  turned  over  to  a  contractor. 
Later,  through  unofficial  sources,  the  test  facility  commander 
found  that  the  Army  had  an  actual  enemy  system,  had  contracted 
with  the  same  contractor  to  build  a  working  replica,  and  provided 
the  contractor  a  far  more  complete  set  of  specifications  than  the 
Foreign  Technology  Division  had  given  the  test  facility. 
Fortunately,  the  test  facility  commander  was  able  to  amend  the 
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contract  and  allow  the  contractor  to  use  the  Army  specifications. 
In  other  cases,  however,  threat  simulators  have  been  built  based 
on  inaccurate  and  incomplete  intelligence  and  have  never  been 
validated . 

The  problem  of  insufficient  intelligence  support  stems  at 
least  partially  from  the  way  the  Defense  Intelligence  Agency 
assigns  intelligence  analysis  responsibilities  to  the  services. 
The  major  criterion  is  to  assign  responsibility  to  the  service 
that  develops  and  operates  a  similar  United  States  system,  and 
not  necessarily  to  the  service  that  the  foreign  weapon  system 
most  seriously  threatens.  Thus,  the  Army  is  assigned  primary 
responsibility  for  surface-to-air  missiles  even  though  they  are  a 
major  threat  to  the  Air  Force. 

It  is  too  simplistic  to  suggest  that  poor  intelligence  sup¬ 
port  is  primarily  due  to  inappropriate  tasking  assignments, 
although  it  is  certainly  one  of  the  major  problems.  Many  other 
factors  are  involved  in  the  quality  of  intelligence  support,  as 
numerous  internal  DOD  studies  have  pointed  out.  They  include 

— data  too  sensitive  to  disseminate, 

— verified  data  not  available, 

—  foreign  weapons  too  complex  and  diverse, 

— unavailable  skilled  manpower,  and 

— misplaced  priorities. 

Major  disagreement  over  the  basic  causes  exists  within  the 
intelligence  community.  For  example,  one  argument  within  the 
Defense  intelligence  Agency  holds  that  overly  restrictive  DOD 
policies  severely  limit  dissemination  of  sensitive  intelligence 
data.  Counter  to  that  argument,  another  faction  within  the 
Defense  Intelligence  Agency  holds  that  it  is  the  absence  of  hard 
data,  not  prohibitions  on  dissemination,  that  lead  to  accusations 
of  poor  intelligence  support. 

Intelligence  support  problems  are  well  known  and  have  been 
documented  by  DOD.  In  an  attempt  to  improve  development  of 
threat  representative  simulators,  DOD  has  a  tri-service  group 
called  the  CROSSBOW-S  committee.  The  committee  reviews  and 
coordinates  development  of  threat  simulators  and  gathers  the 
latest  Central  and  Defense  Intelligence  Agency  estimates  of  the 
threat.  The  committee  has  generated  reliable  information  on 
several  Soviet  systems  that  has  been  used  in  developing  threat 
simulators.  While  the  committee's  charter  and  work  to  date  shows 
good  potential,  our  work  and  other  DOD  studies  show  that 
intelligence  support  problems  continue  to  exist. 
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CONCLUSIONS 


The  increasing  complexity  and  capability  of  United  States 
weapon  systems  as  well  as  the  enemy's  makes  the  task  of  testing 
and  evaluation  of  weapon  systems  extremely  difficult.  Not  only 
must  weapon  developers  build  weapons  that  can  defeat  the  most 
advanced  threats,  the  developer  must  test  its  weapons  with  test 
resources  that  duplicate,  to  the  extent  possible,  the  performance 
of  the  enemy's  weapons.  Only  then  can  the  full  capabilities  and, 
perhaps  more  importantly,  limitations  of  our  weapons  be 
determined . 

Many  barriers  must  be  overcome  to  ensure  the  successful 
development  and  use  of  electronic  warfare  threat  simulators  and 
aerial  targets.  Some,  like  technology,  require  time  and 
continued  effort.  Even  if  a  test  resource  is  successfully 
developed,  there  may  be  other  limitations  on  its  use  that 
constrain  totally  realistic  testing.  Environmental  consid¬ 
erations,  for  example,  prevent  high  powered  jammers  from  being 
used  near  civilian  communications  systems.  Safety  considerations 
may  also  restrict  the  use  of  a  very  low-altitude  target.  How¬ 
ever,  we  believe  improvements  are  possible  through  direct  manage¬ 
ment  action.  These  include 

— earlier  and  more  detailed  test  resource  planning, 

—  improved  organizational  structures, 

—  increased  management  emphasis,  and 

— better  intelligence  support. 

The  services  have  recognized  the  test  resource  planning  and 
management  problems  and  are  beginning  to  take  action.  Most  of 
their  efforts,  however,  identify  problems  but  do  not  offer  plans 
for  corrective  action.  We  are  encouraged  by  their  latest 
efforts,  but  recognize  that  they  are  only  first  steps  and  should 
be  followed  by  definitive  action  plans. 

Although  our  report  focused  on  the  management  and  use  of 
threat  simulators  and  aerial  targets,  we  believe  the  problems  in 
planning  and  management  emphasis  apply  to  testing  and  test 
resources  in  general. 

While  current  DOD  policy  requires  specific  identification  of 
test  resources  very  early  in  the  planning  process,  the  policy  is 
not  being  enforced  by  OSD  or  the  services.  Too  often,  the  ident¬ 
ification  of  test  resources  lacks  specificity  and  timeliness. 
Consequently,  test  resource  designers  and  developers  do  not  have 
adequate  time  to  do  their  job.  in  addition,  the  test  agency's 
ability  to  test  against  a  full  threat  spectrum  and  the  conse¬ 
quences  of  insufficient  test  resources  are  not  surfaced  in  the 


test  and  evaluation  master  plans.  Without  rigorous  early 
planning,  the  necessary  test  resources  will  not  be  on  the  test 
ranges  when  the  weapon  systems  need  them. 

The  Navy  and  the  Air  Force  organizational  structures  for 
threat  simulator  and  aerial  target  development  are  hindering 
rather  than  promoting  the  timely  development  of  these  resources. 
As  a  result,  direct  competition  exists  between  test  resources  and 
weapon  systems.  In  addition,  fragmented  organizational 
structures  result  in  insufficient  coordination,  inadequate  or 
dupli-cate  requirements,  insufficient  support  planning,  and  a 
lack  of  cost  effectiveness  in  the  acquisition,  support,  and  use 
of  test  resources.  Threat  simulators  and  aerial  targets  are  a 
means  to  evaluate  the  combat  worth  of  weapon  systems — they  are 
not  weapons  themselves.  We  believe,  therefore,  that  these 
resources  should  neither  be  considered  nor  managed  as  if  they 
were  weapons.  In  addition,  we  believe  certain  of  the  current 
organizational  arrangements  create  conflicts  of  interest  and 
should  be  eliminated. 

A  DOD-w ide  need  for  hetter  management  support  for  electronic 
warfare  and  aerial  target  test  resources  exists.  Test  resources 
do  not  receive  the  management  emphasis  necessary  to  ensure  their 
timely  development. 

Although  threat  simulators  depend  on  accurate  and  timely 
intelligence  estimates  of  the  threat,  many  development  efforts 
are  hampered  by  insufficient  intelligence  support.  Major  dis¬ 
agreement  within  the  intelligence  community  exists  over  the 
causes  of  and  solutions  to  known  problems.  We  believe  further 
action  is  necessary  to  solve  the  problems. 

AGENCY  COMMENTS  AND  OUR  EVALUATION 


Representatives  from  OSD,  the  Army,  Navy,  and  A:1  r  Force  pro¬ 
vided  official  oral  comments  on  a  draft  of  this  report.  They 
were  primarily  concerned  that  we  had  over  generalized  in  charac¬ 
terizing  all  testing  as  inadequate  and  all  test  resources  as 
deficient,  and  in  stating  that  DOD  is  not  committed  to  a  sound 
test  resource  program.  Our  report  focuses  on  the  management  and 
use  of  two  types  of  test  resources:  electronic  warfare  threat 
simulators  and  aerial  targets.  We  do  not  intend  to  generalize 
our  findings  to  cover  all  testing  and  test  resource  areas.  As  a 
result  of  DOD' s  concerns,  we  made  changes  in  the  report  to  avoid 
such  implications.  In  some  cases,  such  as  early  planning,  our 
findings  cover  the  overall  issue  of  test  resource  planning  and, 
thus,  our  recommendations  are  to  improve  all  test  resource 
planning . 

Overall,  DOD  believes  that  its  testing  program  and  test 
resources  are  adequate,  and  that  it  is  committed  to  a  continuing, 
sound  test  resources  program.  DOD  also  stated  that  our  construc- 
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tive  suggestions  in  threat  simulator  and  aerial  target  test 
resources  are  welcome  and  that  they  will  continue  to  emphasize 
these  two  areas. 

As  discussed  below,  we  remain  convinced  that  testing  in  the 
two  areas  we  reviewed  should  be  improved. 

Complete  versus  adequate  testing 

DOD  disagreed  with  our  finding  that  several  weapon  systems 
will  be  fielded  without  complete  testing  because  existing  aerial 
targets  cannot  adequately  stress  the  systems'  capabilities.  They 
contend  that  although  testing  may  not  be  complete,  it  is 
adequate.  DOD  uses  a  combination  of  actual  tests  and  simulations 
to  provide  the  data  base  for  decisions.  It  remains  a  fact, 
however,  that  existing  aerial  targets  do  not  replicate  the 
capabilities  of  threat  systems  in  such  important  areas  as  speed, 
altitude,  maneuverability,  radar  and  infrared  signatures,  and 
electronic  countermeasures  capabilities.  Laboratory  simulations, 
while  giving  indications  of  performance  in  certain  environments, 
do  not  demonstrate  operational  performance  as  would  an  actual 
firing  at  a  representative  target,  while  we  believe  DOD  obtains 
as  much  data  as  it  can  given  current  resources,  we  also  believe 
that  the  adequacy  and  availability  of  test  resources  needs  to  be 
improved  so  that  the  completeness  and  the  adequacy  of  testing  can 
be  improved.  This,  in  fact,  is  the  primary  message  of  our 
report.  (See  p.  9.) 

Air  Force  management 

DOD  also  disagreed  with  our  finding  that  Air  Force  manage¬ 
ment  of  threat  simulators  is  fragmented.  An  Air  Force  Inspector 
General  report  issued  in  1979  and  an  Air  Force  Systems  Command 
study  of  test  resources  issued  in  1982  corroborates  our  audit 
findings.  We  have  not  changed  the  report.  (See  p.18.) 

Intelligence  support 

DOD  disagreed  with  our  finding  of  inadequate  intelligence 
support  to  the  test  community,  including  test  resource  develop¬ 
ers.  They  cite  a  triservice  coordination  committee,  called 
CROSSBOW-S,  as  serving  to  ensure  realistic  testing  with  realistic 
threat  simulators.  We  are  aware  that  this  committee  has  gener¬ 
ated  reliable  threat  information  for  the  test  community  for  some 
systems.  However,  as  this  report  points  out,  a  large  number  of 
simulator  deficiencies  still  remain  and  many  improvements  can  be 
made . 

When  our  audit  started,  we  did  not  consider  intelligence 
support  to  be  one  of  the  most  critical  issues.  During  the  audit, 
however,  many  members  of  the  test  community,  from  the  test  ranges 
to  the  resource  developers,  complained  about  what  they  perceived 
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as  the  genuinely  poor  quality  of  support  from  the  intelligence 
community.  They  supported  their  concerns  with  specific 
examples.  Their  complaints  were  also  borne  out  by  Naval  Audit 
Service  and  Defense  Audit  Service  reports  that  were  very  critical 
of  intelligence  support  to  the  test  community.  Since  our  audit 
found  that  no  significant  changes  have  occurred,  we  have  not 


changed  our  report.  (See  p.  21.) 

DOD  also  disagreed  with  our  proposal  to  review  all  of  the 
issues  surrounding  intelligence  support,  citing  their 
disagreements  with  our  findings  as  support.  DOD,  however,  has 
not  identified  the  full  extent  of  the  problem  and  unless  a 
critical  review  of  the  problem  and  its  causes  is  undertaken,  the 
poor  support  will  continue.  We  reaffirm  our  proposal.  (See  p. 
27. ) 

Test  planning  process 

DOD  disagreed  with  our  proposals  to  strengthen  the  quality 
and  usefulness  of  the  test  planning  process.  They  agreed  that 
our  suggested  actions  should  occur  but  disagreed  that  the  TEMPis 
the  appropriate  vehicle.  No  other  document,  however,  is  prepared 
or  available  early  enough  in  the  test  planning  process  to  address 
testing  and  surface  issues  to  higher  DOD  management.  In  the 
absence  of  any  DOD  suggested  alternative,  we  have  not  changed  our 
proposals.  (See  p  27.) 

RECOMMENDATIONS  TO  THE  SECRETARY  OF  DEFENSE 

To  improve  the  planning  and  management  of  critical  test 
resources,  we  recommend  that  the  Secretary  of  Defense: 

— Require  the  Director,  Defense  Test  and  Evaluation  to 
enforce  existing  requirements  for  the  preparation  and 
approval  of  weapon  system  TEMPS  before  the  demonstration/ 
validation  and  subsequent  decision  milestones. 

— Require  predemonstration/validation  phase  TEMPs  to 
state  whether  or  not  test  resource  requirements  are 
available,  and  outline  what  actions  have  been  or  need  to 
be  taken  to  develop  or  acquire  those  not  available.  In 
addition,  the  effect  of  being  unable  to  test  against  the 
full  threat  spectrum  should  be  clearly  identified. 

— Require  operational  test  and  evaluation  agencies  to 

state  in  the  initial  TEMP  their  ability  (or  inability)  to 
adequately  assess  a  weapon's  operational  suitability  and 
effectiveness,  given  currently  available  test  resources. 
The  adequacy  of  test  resources  and  the  effect  of 
inadequate  or  incomplete  testing  should  be  clearly  spelled 


— Require  electronic  warfare  and  aerial  target  test  resource 
developers  to  work  closely  with  the  weapon  developers  and 
test  agencies  during  early  test  planning  to  identify  the 
critical  test  resources  needed  to  fully  assess  weapon 
system  effectiveness. 

— Transfer  Navy  and  Air  Force  threat  simulator  and  aerial 
target  acquisition  responsibility  to  an  organization 
separate  from  the  weapon  systems  development  activity. 

The  gaining  organization  should  have,  as  does  the  Army's 
threat  simulator  organization,  the  independence, 
authority,  responsibility,  and  funds  to  ensure  the 
successful  acquisition  of  test  resources. 

— Establish  a  joint-service  threat  simulator  and  aerial 
target  improvement  program  to  identify,  time  phase,  and 
prioritize  DOD-wide  test  resource  deficiencies;  and  start 
development  of  the  resources  necessary  to  match  the  test 
capability  with  current  requirements. 

— Initiate  a  review  of  intelligence  support  to  identify  the 
underlying  causes  and  to  solve  the  problems  of  inadequate 
support  to  the  threat  simulator  development  community.  In 
particular,  the  appropriateness  of  Defense  Intelligence 
Agency  assignments  to  the  service  intelligence  organiza¬ 
tions  and  the  capability  of  those  organizations  to 
support  both  weapon  designers  and  the  test  community 
should  be  examined  and  changes  made  where  appropriate. 

RECOMMENDATION  TO  THE  CONGRESS 

This  report  shows  that  major  weapon  systems  are  being 
deployed  without  the  best  possible  demonstration  of  their 
capabilities  under  combat  type  conditions.  DOD  has  been  giving 
the  Congress  reports  on  the  cost,  schedule,  and  performance 
status  of  its  major  weapon  systems  since  1969.  Additionally, 
annual  hearings  on  weapon  systems  have  continually  highlighted 
weapon  performance  issues  and  the  need  for  better  test  and 
evaluation,  as  have  many  of  our  reports.  Nevertheless,  the 
problems  associated  with  inadequate  and  incomplete  testing 
continue . 

Since  improvements  in  test  resources  will  lead  to  better 
testing  and  thus,  to  better  weapon  system  performance,  we  recom¬ 
mend  the  Congress  monitor  DOD's  implementation  of  (1)  the 
joint-service  aerial  target  and  electronic  warfare  test  resource 
improvement  program  and  (2)  the  separation  of  test  resource  and 
weapon  system  acquisition  management  in  the  Navy  and  the  Air 
Force . 
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DESCRIPTION  OF  TERMS  USED  IN  THIS  REPORT 
Electronic  warfare 

Electronic  warfare  is  any  military  action  involving  the  use 
of  electromagnetic  energy  to  determine,  exploit,  reduce,  or  pre¬ 
vent  the  enemy’s  use  of  the  electromagnetic  spectrum.  The  use  of 
electronic  warfare  includes  all  levels  and  areas  of  combat, 
including  communications,  radar  systems,  and  electronic  weapons 
guidance  systems.  Since  most  of  todays  sophisticated  weapon  sys¬ 
tems  use,  in  one  way  or  another,  the  electromagnetic  spectrum, 
their  effectiveness  can  he  severely  degraded  in  an  electronic 
warfare  environment. 

Electronic  warfare  can  be  divided  into  three  areas: 

1.  Electronic  warfare  support  measures — actions  taken  to 
search  for,  intercept,  identify,  and  locate  sources  of 
radiated  electromagnetic  energy. 

2.  Electronic  countermeasures — actions  taken  to  prevent  or 
reduce  an  enemy's  use  of  the  electromagnetic  spectrum. 
The  two  primary  means  of  electronic  countermeasures  are 
jamming  and  deception.  Jamming  makes  an  enemy's 
electronic  equipment  unusable.  (See  fig.  1.). 

Deception,  on  the  other  hand,  uses  the  enemy's 
electronic  equipment  to  deceive  or  mislead  through 
manipulation  of  the  enemy's  signals. 

3.  Electronic  counter-countermeasures — actions  taken  to 
overcome  the  enemy's  use  of  electronic  countermeasures. 

Threat  simulators 

Testing  electronic  warfare  equipment  and  evaluating  its 
effectiveness  requires  the  use  of  equipment  that  simulates,  to 
varying  degrees,  the  enemy's  weapons.  Threat  simulators  fall 
into  three  categories  and  range  from  computer  generated  signals 
in  a  laboratory  to  actual  captured  enemy  equipment.  The 
categories  of  threat  simulators  are  emitters,  emulators,  and 
replicas. 

Emitter — reproduces  some  or  all  of  the  threat  systems' 
transmitter  characteristics  (e.g.,  radio  frequency,  pulse  width, 
power  levels,  etc,). 
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(Source:  Air  Force  Phamplet  51-3,  "Electronic  Warfare  Principles") 


Effective  Jamming  Can  Hide  A  Target  And  Make  A  Radar  Useless 


FIGURE  1 
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Emulator — in  addition  to  the  transmitter,  an  emulator  repro¬ 
duces  some  of  the  threat  systems'  receiver  and/or  signal  proces¬ 
sor  characteristics.  In  some  cases,  an  emulator  may  he  a  United 
States  or  allied  system  used  as  a  surrogate  or  modified  to 
closely  resemble  the  threat.  An  emulator,  like  an  emitter,  is 
not  required  to  physically  resemble  the  threat  system. 

Replica — a  simulator  which  possesses  all  relevant  electronic 
and  physical  properties  of  the  threat  system.  (See  figs.  2  and 
3.  ) 

TEST  AND  EVALUATION 

Test  and  evaluation  is  conducted  throughout  the  weapon  sys¬ 
tems  acquisition  process  to  identify  and  reduce  development  and 
production  risks  and  to  estimate  how  well  a  system  performs  in 
its  intended  environment.  Test  and  evaluation  is  normally 
divided  into  three  categories. 

Development  test  and  evaluation  verifies  that  the  weapon 
meets  its  technical  performance  specifications  and  objectives. 
Testing  is  usually  done  by  the  weapon  system  developer. 

Operational  test  and  evaluation  estimates  a  systems  opera¬ 
tional  suitability  and  effectiveness.  In  other  words,  it  deter¬ 
mines  how  well  a  system  can  perform  its  mission  and  be  maintained 
under  operating  conditions. 

production  acceptance  test  and  evaluation  demonstrates  that 
the  weapon,  after  it  has  been  produced,  meets  the  requirements 
and  specifications  of  the  procurement  contract. 


30 


► 

L« 


"fl 

APPENDIX  II 

APPENDIX 

.  V. 

■ 

OUR  REPORTS  DEALING  WITH 

TEST  AND  EVALUATION 

Report 

B 

Title/subject 

number 

Date 

Adverse  Effects  of  Large  Scale 
production  of  Major  Weapons 

Before  Completion  of  Development 
and  Testing 

B-163058 

11/19/70 

1 

■ 

i 

The  Importance  of  Testing  and 
Evaluation  in  the  Acquisition 

Process  for  Major  Weapon 

Systems 

B- 16 30 58 

8/07/72 

r* 

L 

Improvements  Needed  in  Development 
Testing 

B-163058 

3/07/74 

* 

Review  of  Testing  and  Evaluation 
Policies  and  Procedures 

B-163058 

4/18/74 

Review  of  the  Adequacy  of 

Department  of  Defense  Test 

Resources 

PSAD-75-84 

4/30/75 

► 

n 

Effectiveness  of  Testing  of 

Selected  Major  Weapon  Systems 

PSAD-75-74 

6/04/75 

Does  the  Department  of  Defense 

Have  More  Test  Capacity  Than 

It  Needs? 

PSAD-76-75 

3/01/76 

Report  to  the  Secretary  of  Defense 
on  Operational  Testing  on  the  Major 
Caliber  Lightweight  Gun 

PSAD-77-4 

11/05/76 

i 

Navy  Operational  Test  and 

Evaluation:  A  Valuable  To<-  ’ 

Not  Fully  Utilized 

PSAD-78-77 

3/29/78 

Operational  Testing  of  Air  Force 
Systems  Requires  Several 

Improvements 

PSAD-78-102 

6/02/78 

M 


t 
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Report 

Title/subject  number  Date 

Report  to  the  Secretary  o£  Defense  PSAD-78-131  7/25/78 

on  Operational  Test  and  Evaluation 
of  Foreign  Built  Systems 

Report  to  the  Secretary  of  Defense  PSAD-79-1  10/19/78 

on  Follow-On  Operational  Test  and 
Evaluat ion 

Need  for  More  Accurate  Weapon  PSAD-79-46  3/08/79 

System  Test  Results  to  be  Reported 
to  the  Congress 

Report  to  the  Secretary  of  Defense  PSAD-79-86  6/25/79 

on  Development  Test  and  Evaluation 
of  Six  Systems 

Army  Operational  Test  and  C-PSAD-80-2  11/13/79 

Evaluation  Needs  improvement 

DOD  Information  provided  to  the  C-PSAD-80-24  5/09/80 

Congress  on  Major  Weapon  Systems 
Could  he  More  C  :>mplete  and 
Useful 

Report  to  the  Secretary  of  GAO/MASAD-82-38  8/06/82 

Defense  on  use  of  the  Design 

for  Testability  Concept  in  the 

Development  and  Acquisition  of 

Major  Systems 

Note :  This  listing  does  not  include  the  numerous  individual 

weapon  system  reports  issued  over  the  years  for  use  by 
the  Congress;  many  of  these  reports  deal  with  issues  of 
system  performance  and  involve  test  and  evaluation. 
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DESCRIPTION  OF  WEAPON  SYSTEMS  INCLUDED  IN  THIS  REPORT 


SINGLE  CHANNEL  GROUND  AND  AIRBORNE  RADIO  SYSTEM 


The  Very  High  Frequency  Single  Channel  Ground  and  Airborne 
Radio  System  is  the  Army's  future  combat  radio.  It  will  be  the 
primary  means  of  communication  for  armor,  artillery,  and  infantry 
forces.  The  radio  will  he  configured  for  aircraft,  vehicular, 
and  manpack  applications. 

Advanced  development  began  in  1978  and  the  production  award 
is  scheduled  for  July  1983.  The  total  estimated  cost,  as  of 
September  1982,  is  $531  million  for  38,000  radios. 

TACTICAL  ARMY  COMMUNICATIONS  JAMMING  SYSTEM 


The  Tactical  Army  Communications  Jamming  System  is  a 
ground-mobile  communications  jammer  used  at  division  and  corps 
levels.  The  system  can  jam  enemy  tactical  communications  and 
digital  (secure)  voice  and  data  links.  The  system  is  in  produc¬ 
tion  and  is  scheduled  for  followup  operational  testing  during  the 
third  quarter,  fiscal  year  1983.  Total  estimated  program  cost  as 
of  September  1982,  is  $240.3  million. 


PATRIOT 

The  Patriot  Air  Defense  Missile  System  (formerly  the  SAM-D) 
development  program  began  in  1965.  Its  mission  is  to  provide 
low-  and  medium-altitude  air  defense  coverage  to  the  field  Army. 
The  Patriot  will  replace  the  Hawk  and  the  Nike  Hercules  weapon 
systems . 

The  system  was  approved  for  limited  production  in  September 
1980.  The  first  Patriot  battalion  was  activated  in  May  1982. 

As  of  September  1982,  total  program  cost  is  estimated  to  be  $11.3 
hill  ion . 


HIGH  SPEED  ANTI-RADIATION  MISSILE 

The  High  Speed  Anti-Radiation  Missile  is  being  developed 
jointly  by  the  Navy  and  the  Air  Force  to  give  aircraft  performing 
surface  attack  missions  a  better  chance  of  penetrating  enemy 
radar  defenses  by  destroying  or  suppressing  land-  and  sea-based 
radars  of  enemy  air  defense  systems. 
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The  missile  is  in  limited  production  and  undergoing  opera¬ 
tional  test  and  evaluation.  As  of  September  1982,  the  Air 
Force’s  estimated  total  program  cost  was  $4.7  billion.  The 
Navy's  estimated  total  program  cost  was  $3.1  hillion  for  a  total 
joint  program  cost  of  $7.8  billion. 

AN/ALQ-126B 

The  AN/ALQ-126B  Defensive  Electronic  Countermeasures  Set  is 
a  follow-on  to  the  AN/ALQ-126A  and  was  developed  to  increase  pro¬ 
tection  for  naval  aircraft  against  modern,  diversified  radar 
controlled  weapon  systems  such  as  land-  and  sea-based  surface-to- 
air  missile  systems.  As  of  September  1982  the  total  program  cost 
was  estimated  at  $725  million. 

The  AN/ALQ-126B  completed  its  operational  evaluation  in 
April  1982. 

PHOENIX  ( AIM— 54C ) 

The  Phoenix  is  an  all-weather  air-to-air  missile  to  counter 
high-  and  low-altitude  aircraft  and  missiles.  It  has  the  capa¬ 
bility  to  engage  multiple  targets,  and  operate  at  both  visual  and 
beyond  visual  ranges.  The  F-14A  is  the  only  aircraft  capable  of 
carrying  the  phoenix  missile. 

The  "C"  model  development  program  began  in  1976  to  provide 
increased  lethality,  electronic  counter-countermeasures  perform¬ 
ance,  high-  and  low-altitude  performance,  and  improved  relia¬ 
bility  and  maintainability. 

Full-scale  production  is  expected  to  begin  shortly.  A  total 
of  2,680  missiles  are  scheduled  to  be  built  through  fiscal  year 
1988.  As  of  June  30,  1982,  the  Navy  estimated  that  the  total 
program  cost  would  he  $3.1  billion. 

B-52  G/H  OFFENSIVE  AVIONICS  SYSTEM 


The  B-52  G/H  Offensive  Avionics  System  is  intended  to  pro¬ 
vide  improved  reliability  and  maintainability  to  the  B-52  G/H 
hombing  and  navigation  system  which  is  becoming  logistically 
unsupportahle .  It  allows  the  B-52  G/H  to  carry,  align,  target, 
and  launch  the  AGM-86B  Air-Launched  Cruise  Missile.  The  program, 
started  in  1978,  was  approved  for  production  in  July  1979  but  did 
not  start  development  test  and  evaluation  and  initial  operational 
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test  and  evaluation  until  June  1980.  That  phase  of  testing  was 
completed  in  September  1982.  A  separate  phase  of  testing,  called 
Integrated  Weapon  System  testing  that  involves  the  B-52,  the  Air- 
Launched  Cruise  Missile,  and  the  Short-Range  Attack  Missile, 
hegan  in  October  1981,  and  will  continue  until  December  1983. 
Initial  operational  capability  was  achieved  in  December  1982. 
Total  projected  program  costs  are  $3.6  billion  (as  of  September 
1982). 

F-l 6  MULTINATIONAL  STAGED  IMPROVEMENT  PROGRAM 


The  F-l 6  Multinational  Staged  Improvement  Program  is  a  three 
stage  approach  to  upgrade  the  capabilities  of  the  F-16  multimis¬ 
sion  fighter  aircraft.  Extending  from  1980  to  the  1990s,  the 
program  will  incorporate  improvements  in  weaponry,  communica¬ 
tions,  navigation,  and  sensors,  at  a  projected  cost  of  $3.9  bil¬ 
lion  for  the  first  144  aircraft  (as  of  September  1982). 

Stage  I  of  the  program  is  underway,  fitting  new  aircraft 
with  structural,  wiring,  and  cooling  system  changes  to  support 
future  growth.  Stage  II  has  begun  with  contractor  testing,  and 
is  expected  to  continue  until  1984  with  primarily  avionics, 
radar,  and  cockpit  changes.  Stage  III,  scheduled  to  extend 
several  years  further  into  the  future,  will  involve  progressive 
integration  of  such  advanced  systems  as  the  AMRAAM,  the  Airborne 
Self-Protection  jammer,  the  Precision  Location  Strike  System,  and 
the  Low-Altitude  Navigation  and  Targeting  infrared  System  for 
Night. 

EF-111A 

The  EF-111A  Tactical  Jamming  System  is  an  Air  Force  F-111A 
aircraft  modified  to  carry  radar  jamming  systems.  It  will  be 
used  to  protect  friendly  aircraft  by  jamming  enemy  early  warning, 
acquisition,  and  ground  control  intercept  radars.  The  EF-111A  is 
replacing  the  EB-66  weapon  system  which  has  been  retired. 

Full-scale  production  began  in  March  1979.  As  of  June  30, 
1982,  the  Air  Force  estimated  that  total  program  costs  would  be 
$1.5  hillion  for  42  aircraft. 
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LOCATIONS  VISITED 


Office  of  the  secretary  of  Defense: 

Office  of  the  Under  Secretary 

of  Defense,  Research  and  Engineering 
Washington,  D.C. 

Assistant  Deputy  Under  Secretary  for  Acquisition 
Director,  Major  Systems  Acquisition 
Washington,  D.C. 

Deputy  Under  Secretary  (C3I) 

Director,  Electronic  Warfare  and 
C3  Countermeasures 
Washington,  D.C. 

Director,  Defense  Test  and  Evaluation,  Washington,  D.C. 
Defense  Audit  Agency,  Washington,  D.C. 

Defense  Science  Board,  Washington,  D.C. 

Defense  intelligence  Agency,  Washington,  D.C. 

Department  of  the  Army: 

Deputy  Chief  of  Staff  for  Operations  and  plans 
Washington,  D.C. 

Deputy  Chief  of  staff  for  Research, 

Development,  and  Acquisition 
Washington,  D.C. 

Army  Audit  Agency,  Washington,  D.C. 

U.S.  Army  Operational  Test  and  Evaluation  Agency 
Falls  Church,  Virginia 

White  Sands  Missile  Range 
White  Sands,  New  Mexico 

Office  of  Missile  Electronic  Warfare 
White  Sands,  New  Mexico 

Army  Defense  School 
ADATS  Program  office 
Fort  Bliss,  Texas 
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Electronic  Proving  Ground 
Fort  Huachuca,  Arizona 

U.S.  Army  Test  and  Evaluation  Command 
Aberdeen  Proving  Ground,  Maryland 

Department  of  the  Navy: 

Director,  Command  and  Control 
Electronic  Warfare  Division 
Washington,  D.C. 

Office  of  Research,  Development,  Test 
and  Evaluation 
Washington,  D.C. 

Naval  Air  Systems  Command,  Washington,  D.C. 
F-14/Phoenix  project  Office 
Defense  Suppression  Project  Office 
Reconnaissance,  Electronic  Warfare, 

Special  Operations,  and  Naval 
Intelligence  (REWSON)  Project  Office 
Assistant  Commander  for  Test  and  Evaluation 

Naval  Electronic  Systems  Command,  Washington,  D.C. 
REWSON  Systems  Project  Office 

Naval  Audit  Service,  Washington,  D.C. 

Headquarters,  Navy  Operational 
Test  and  Evaluation  Force 
Norfolk,  Virginia 

Deputy  Commander,  Operational 
Test  and  Evaluation  Force 
San  Diego,  California 

Naval  weapons  Center 
China  Lake,  California 

Air  Test  and  Evaluation  Squadron  Five 
China  Lake,  California 

Pacific  Missile  Test  Center 
Point  Mugu,  California 

Air  Test  and  Evaluation  Squadron  Four 
Point  Mugu,  California 
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U.S.  Pacific  Fleet,  San  Diego,  California 
Headquarters,  Surface  Forces 
Headquarters,  Naval  Air  Force 

Department  of  the  Air  Force: 

Deputy  Chief  of  Staff,  Plans  and  Operations 
Washington,  D.C. 

Deputy  Chief  of  Staff,  Research  Development 
and  Acquisition 
Washington,  D.C. 

Air  Force  Audit  Agency,  Washington,  D.C. 

Air  Force  Test  and  Evaluation  Center 
Albuquerque,  New  Mexico 

Headquarters,  Strategic  7  Lr  Command 
Offutt  Air  Force  Base,  Nehraska 

Headquarters,  Tactical  Air  Command 
Langley,  Air  Force  Base,  Virginia 

Aeronautical  Systems  Division 
Wright-patterson  Air  Force  Base,  Ohio 

Armament  Division 

Eglin  Air  Force  Base,  Florida 

Air  Force  Electronic  Warfare  Evaluation  Simulator, 

Fort  Worth,  Texas 

Joint  Electronic  Warfare  Center 
San  Antonio,  Texas 

Air  Defense  Weapons  Center 
Tyndall  Air  Force  Base,  Florida 

Other: 

American  Defense  Preparedness  Association,  Washington,  D.C. 
National  Security  Industrial  Association,  Washington,  D.C. 
General  Dynamics  Convair  Division,  San  Diego,  California 


